Research Data Explored: Citations versus Altmetrics

The study explores the citedness of research data, its distribution over time and how it is related to the availability of a DOI (Digital Object Identifier) in Thomson Reuters’ DCI (Data Citation Index). We investigate if cited research data “impact” the (social) web, reflected by altmetrics scores, and if there is any relationship between the number of citations and the sum of altmetrics scores from various social media-platforms. Three tools are used to collect and compare altmetrics scores, i.e. PlumX, ImpactStory, and Altmetric.com. In terms of coverage, PlumX is the most helpful altmetrics tool. While research data remain mostly uncited (about 85%), there has been a growing trend in citing data sets published since 2007. Surprisingly, the percentage of the number of cited research data with a DOI in DCI has decreased in the last years. Only nine repositories account for research data with DOIs and two or more citations. The number of cited research data with altmetrics scores is even lower (4 to 9%) but shows a higher coverage of research data from the last decade. However, no correlation between the number of citations and the total number of altmetrics scores is observable. Certain data types (i.e. survey, aggregate data, and sequence data) are more often cited and receive higher altmetrics scores. Conference Topic Altmetrics Citation and co-citation analysis Introduction Recently, data citations have gained momentum (Piwowar & Chapman, 2010; Borgman, 2012; Torres-Salinas, Martín-Martín, & Fuente-Gutiérrez, 2013). This is reflected, among others, in the development of data-level metrics (DLM), an initiative driven by PLOS, UC3 and DataONE 1 , to track and measure activity on research data, and the recent announcement of CERN to provide DOIs for each dataset they share through their novel Open Data portal 2 . Data citations are citations included in the reference list of a publication that formally cite either the data that led to a research result or a data paper 3 . Thereby, data citations indicate the influence and reuse of data in scientific publications. First studies on data citations showed that certain well-curated data sets receive far more citations or mentions in other articles than many traditional articles (Belter, 2014). Citations, however, are used as a proxy for the assessment of impact primarily in the “publish or perish” community; to consider other disciplines and stakeholders of research, such as industry, 1 http://escholarship.org/uc/item/9kf081vf 2 https://www.datacite.org/news/cern-launches-data-sharing-portal.html 3 http://www.asis.org/Bulletin/Jun-12/JunJul12_MayernikDataCitation.html government and academia, and in a much broader sense, the society as a whole, altmetrics (i.e. social media-based indicators) are emerging as a useful instrument to assess the “societal” impact of research data or at least to provide a more complete picture of research uptake, besides more traditional usage and citation metrics (Bornman, 2014; Konkiel, 2013). Previous work on altmetrics for research data has mainly focused on motivations for data sharing, creating reliable data metrics and effective reward systems (Costas et al., 2012). This study contributes to the research on data citations in describing their characteristics as well as their impact in terms of citations and altmetrics scores. Specifically, we tackle the following research questions:  How often are research data cited? Which and how many of these have a DOI? From which repositories do research data originate?  What are the characteristics of the most cited research data? Which data types and disciplines are the most cited? How does citedness evolve over time?  To what extent are cited research data visible on various altmetrics channels? Are there any differences between the tools used for altmetrics scores aggregation? Data sources On the Web, a large number of data repositories are available to store and disseminate research data. The Thomson Reuters Data Citation Index (DCI), launched in 2012, provides an index of high-quality research data from various data repositories across disciplines and around the world. It enables search, exploration and bibliometric analysis of research data through a single point of access, i.e. the Web of Science (Torres-Salinas, Martín-Martín & FuenteGutiérrez, 2013). The selection criteria are mainly based on the reputation and characteristics of the repositories 4 . Three document types are available in the DCI: data set, data study, and repository. The document type “repository” can distort bibliometric analyses, because repositories are mainly considered as a source, but not as a document type. First coverage and citation analyses of the DCI have been performed April-June 2013 by the EC3 bibliometrics group of Granada (Torres-Salinas, Jimenez-Contreras & Robinson-Garcia, 2014; Torres-Salinas, Robinson-Garcia & Cabezas-Clavijo, 2013). They found that data is highly skewed: Science areas accounted for almost 80% of records in the database and four repositories contained 75% of all the records in the database; 88% of all records remained uncited. In Science, Engineering and Technology citations are concentrated among datasets, whereas in the Social Sciences and Arts & Humanities, citations often refer to data studies. Since these first analyses, DCI has been constantly growing, now indexing nearly two million records from high-quality repositories around the world. One of the most important enhancements of the DCI has undoubtedly been the inclusion of “figshare 5 ” as new data source which led to an increase of almost a half million of data sets and 40.000 data studies (i.e. about one fourth of the total coverage in the database). Gathering altmetrics data is quite laborious since they are spread over a variety of social media platforms which each offer different applications programming interfaces (APIs). Tools, which collect and aggregate these altmetrics data come in handy and are now fighting for market shares since also large publishers increasingly display altmetrics for articles (e.g., 4 http://thomsonreuters.com/data-citation-index, http://thomsonreuters.com/products/ip-science/04_037/dciselection-essay.pdf 5 http://figshare.com Wiley 6 ). There are currently three big altmetrics data providers: ImpactStory 7 , Altmetric.com, and PlumX 8 . Whereas Altmetrics.com and PlumX focus more on gathering and providing data for institutions (e.g., publishers, libraries, or universities), ImpactStory’s target group is the individual researcher who wants to include altmetrics information in her CV. ImpactStory is a web-based tool, which works with individually assigned permanent identifiers (such as DOIs, URLs, PubMed IDs) or links to ORCID, Figshare, Publons, Slideshare, or Github to auto-import new research outputs like e.g. papers, data sets, slides. Altmetric scores from a large range of social media-platforms, including Twitter, Facebook, Mendeley, Figshare, Google+, and Wikipedia 9 , can be downloaded as .json or .csv (as far as original data providers allow data sharing). With Altmetric.com, users can search within a variety of social media-platforms (e.g., Twitter, Facebook, Google+, or 8,000 blogs 10 ) for keywords as well as for permanent identifiers (e.g., DOIs, arXiv IDs, RePEc identifiers, handles, or PubMed IDs). Queries can be restricted to certain dates, journals, publishers, social media-platforms, and Medline Subject Headings. The search results can be downloaded as .csv from the Altmetric Explorer (web-based application) or via the API. Plum Analytics or Plum X (the fee-based altmetrics dashboard) offers article-level metrics for so-called artifacts, which include articles, audios, videos, book chapters, or clinical trials 11 . Plum Analytics works with ORCID and other user IDs (e.g., from YouTube, Slideshare) as well as with DOIs, ISBNs, PubMed-IDs, patent numbers, and URLs. Because of its collaboration with EBSCO, Plum Analytics can provide statistics on the usage of articles and other artifacts (e.g., views to or downloads of html pages or pdfs), but also on, amongst others, Mendeley readers, GitHub forks, Facebook comments, and YouTube subscribers.